Trajectory prediction on top-down scenes

ABSTRACT

Techniques are discussed for determining predicted trajectories based on a top-down representation of an environment. Sensors of a first vehicle can capture sensor data of an environment, which may include agent(s) separate from the first vehicle, such as a second vehicle or a pedestrian. A multi-channel image representing a top-down view of the agent(s) and the environment and comprising semantic information can be generated based on the sensor data. Semantic information may include a bounding box and velocity information associated with the agent, map data, and other semantic information. Multiple images can be generated representing the environment over time. The image(s) can be input into a prediction system configured to output a heat map comprising prediction probabilities associated with possible locations of the agent in the future. A predicted trajectory can be generated based on the prediction probabilities and output to control an operation of the first vehicle.

BACKGROUND

Prediction techniques can be used to determine future states of entities in an environment. That is, prediction techniques can be used to determine how a particular entity is likely to behave in the future. Current prediction techniques often involve physics-based modeling or rules-of-the-road simulations to predict future states of entities in an environment.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

FIG. 1 is a pictorial flow diagram of an example process for capturing sensor data, inputting images representing a top-down view of the environment into a prediction system, receiving heat maps associated with prediction probabilities, and generating a predicted trajectory based at least in part on the prediction probabilities, in accordance with embodiments of the disclosure.

FIG. 2 depicts a block diagram of an example system for implementing the techniques described herein.

FIG. 3 is a pictorial flow diagram of an example process for receiving heat maps associated with prediction probabilities, determining a predicted point based at least in part on a highest probability in each heat map, and generating one or more predicted trajectories based at least in part on a plurality of predicted points, in accordance with embodiments of the disclosure.

FIGS. 4A and 4B are example illustrations of inputs to the prediction system to determine one or more predicted trajectories, in accordance with embodiments of the disclosure.

FIG. 5 is an illustration of various predicted trajectories as output by a prediction system, in accordance with embodiments of the disclosure.

FIG. 6 depicts an example process for generating predicted trajectories based at least in part on a top-down representation of an environment including information associated with an agent in the environment, in accordance with embodiments of the disclosure.

DETAILED DESCRIPTION

This disclosure is directed to determining predicted trajectories based at least in part on a top-down representation of an environment. For example, sensors of an autonomous vehicle can capture sensor data of an environment, which may include agents separate from the autonomous vehicle, such as other vehicles or pedestrians. An image representing the environment from a top-down perspective can be generated based at least in part on the sensor data. In some examples, the image may include semantic information of the environment and/or the agent. With respect to the agent, such semantic information may comprise a bounding box having extents (e.g., identifying the bounding box as being associated with a vehicle), movement information associated with the bounding box (e.g., velocity, acceleration, etc.), classification information (whether the box represents a vehicle, pedestrian, bicyclist, motorcyclist, etc.). With respect to the environment, the image may comprise semantic information indicative of speed limits, lane widths, lane ends, stop lines, traffic lights, traffic light states, and the like as described in more detail herein.

A plurality of images can be generated representing the environment over time. The image or the plurality of images can be input into a prediction system (e.g., a machine learning model) configured to output a heat map comprising prediction probabilities associated with possible locations of the agent in the future. One or more predicted trajectories can be generated based on the prediction probabilities and output to a planning system to control an operation of the autonomous vehicle.

Sensor data captured by the autonomous vehicle can include LIDAR data, RADAR data, image data, time of flight data, SONAR data, and the like. In some cases, the sensor data can be provided to a perception system configured to determine a type of an agent (e.g., vehicle, pedestrian, bicycle, animal, parked car, tree, building, and the like) in the environment. Further, the perception system can determine, based on the sensor data, movement information about the agent in the environment. The sensor data and any data based on the sensor data can be represented in a top-down view of the environment. For example, the image can represent the agent as a two-dimensional bounding box representing the location of the agent in the environment as well as an extent of the agent (e.g., the length and width of the agent), and a classification of the agent (e.g., vehicle, pedestrian, and the like). Movement information such as velocity information can be represented as a velocity vector associated with the bounding box, although other representations are contemplated.

In some cases, the image to be input into the prediction system can be represented by individual channels of a multichannel image where each channel represents different information about the agent and the environment in which the agent is located. For example, channels may represent, but are not limited to, one or more of: road network information (e.g., semantic labels indicating lanes, crosswalks, stop lines, lane dividers, stop signs, intersections, traffic lights, and the like), traffic light status (e.g., red light, yellow light, green light, etc.), a bounding box associated with the agent, a velocity of the agent in an x-direction and a y-direction, an acceleration of the agent in an x-direction and a y-direction, a blinker status of the agent (e.g., left-turn, right-turn, braking, reverse, hazards, no lights, etc.), and the like. In some examples, the plurality of channels can be input to the prediction system to generate at least one predicted trajectory. In some examples, the image can be input directly to the prediction system (e.g., without generating a plurality of channels based on the image) to generate the at least one predicted trajectory.

In some examples, the number of channels may be reduced into fewer channels. As a non-limiting example, a simple 3-channel (e.g., RGB) image may be used to represent multiple pieces of semantic information. In such an example, a red channel, for instance, may be used to denote a bounding box and the intensity of the red value may be associated with a velocity, acceleration, or the like. Further in such an example, a second channel may encode lane width with intensity indicative of a speed limit, and the like.

In some examples, the prediction system can include a machine learning model trained to output data that can be used to generate one or more predicted trajectories. For example, the machine learning model can output coordinates (e.g., x-coordinates and y-coordinates) associated with the agent (e.g., a third-party vehicle) at one or more times in the future (e.g., 1 second, 2 seconds, 3 seconds, etc.). In some examples, the machine learning model can output coordinates associated with the agent as well as probability information associated with each coordinate. In some examples, such probability information can be represented as an ellipse or other polygon associated with a threshold probability level (e.g., a 65% probability that a location of the agent at a particular time is within the area represented by the ellipse). A predicted trajectory can be generated by interpolating between the coordinates output by the machine learning model. In some examples, the machine learning model can include a convolutional neural network (CNN), which may include one or more recurrent neural network (RNN) layers, such as, but not limited to, long short-term memory (LSTM) layers. In some examples, the machine learning model can output a heat map associated with prediction probabilities. In some examples, at least one predicted trajectory can be determined based at least in part on the heat map.

A heat map can represent a discretized region of the environment proximate to the autonomous vehicle. For example, the heat map can represent a 64×64 grid (or J×K sized grid) representing a 100 meter by 100 meter region around the autonomous vehicle. Of course, the heat map can represent any size of region and can represent any number of discrete portions of the region. In some cases, a portion of the heat map can be referred to as a cell of the heat map. Each cell can comprise a prediction probability representing a probability that the agent will be at the corresponding location in the environment at the time corresponding to the heat map. In some instances, the machine learning model can output a plurality of heat maps, wherein a heat map of the plurality of heat maps can represent probability predictions associated with the agent at a particular time in the future (e.g., 0.5 second, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.).

To determine a predicted trajectory from the heat maps, each heat map can be analyzed individually to determine a highest probability associated with a cell of the heat map. In some cases, the region of the heat map associated with the highest probability can be referred to as a predicted point. In some examples, a predicted point can be selected based at least in part on the highest probability point and one or more costs and/or constraints associated vehicle dynamics of a resulting predicted trajectory. By way of example, if the prediction system outputs five heat maps, the techniques discussed herein can include determining at least five prediction points. A first predicted point can be selected based on corresponding to the highest probability region in the first heat map. A second predicted point can be selected based on a highest probability region in the second heat map and a cost associated with a trajectory between the first predicted point and the second predicted point. Such a cost or constraint may be used, for example, to enforce physical correctness (e.g., to ensure the predicted trajectory follows a physically plausible trajectory). The predicted points can be used to determine a predicted trajectory. In some cases, the predicted trajectory can be determined by interpolating between the points or fitting a curve to the points (e.g., fitting one or more of a polynomial curve, a Bezier curve, a clothoid curve, etc.). In some cases, generating a predicted trajectory and/or selecting the predicted points can be based at least in part on one or more costs and/or constraints associated with vehicle dynamics to prevent or reduce an occurrence where the predicted trajectory represents an unreasonable trajectory (e.g., involving “teleportation” of the vehicle where subsequent points comprise physically impossible motion of the vehicle). Examples of such costs may include, but are not limited to, a velocity cost (e.g., a constant velocity cost), an acceleration cost, an expectation that the agent may follow rules of the road, and the like.

As can be understood, the heat maps may comprise prediction probabilities that may represent a plurality of predicted trajectories for the agent in the environment. In the context of a vehicle traversing an environment, a first predicted trajectory can represent the vehicle making a left-turn through the intersection, while a second predicted trajectory can represent the vehicle going straight through the intersection. In a case where the first predicted trajectory has a higher probability than the second predicted trajectory (e.g., because the sensor data may have captured a left-turn indicator (e.g., a blinker or turn signal) of the vehicle), the operation can include masking, covering, or otherwise removing prediction probabilities of the heat maps that correspond to the first predicted trajectory. Next, the masked heat map may be normalized (e.g., the prediction probabilities can be scaled between 0 and 1) and the highest probability of the masked heat map can be determined as a prediction point. A second trajectory can be based at least in part on the prediction points associated with the masked heat map. That is, the second set of prediction points can be used to generate the second predicted trajectory by evaluating one or more cost functions to determine the second trajectory. This masking process and determining of predicted trajectories can be repeated until a probability of a trajectory does not meet or exceed a prediction threshold. The at least one predicted trajectory can be provided to a planning system of the autonomous vehicle whereby the autonomous vehicle can be controlled based at least in part on the at least one predicted trajectory. In at least other examples, all possible predicted trajectories and/or their corresponding uncertainties can be output to such a planning system.

In some instances, one or more heat maps can be provided to a planning system instead of or in addition to the at least one predicted trajectory.

The techniques discussed herein can improve a functioning of a computing device in a number of additional ways. In some cases, representing the environment and the agent(s) in the environment as a top-down view can represent a simplified representation of the environment for the purposes of generating at least one predicted trajectory. In some cases, the image can represent the environment without extracting particular features of the environment, which may simplify the generation of the prediction system and subsequent generation of the at least one predicted trajectory. In some cases, generating the at least one predicted trajectory can be provided to a planner system of an autonomous vehicle, which may allow the autonomous vehicle to generate more accurate and/or safer trajectories for the autonomous vehicle to traverse an environment. For example, a predicted trajectory suggesting a likelihood of a collision or a near-collision may allow the autonomous vehicle to alter a trajectory (e.g., change lanes, stop, etc.) in order to safely traverse the environment. These and other improvements to the functioning of the computer are discussed herein.

The techniques described herein can be implemented in a number of ways. Example implementations are provided below with reference to the following figures. Although discussed in the context of an autonomous vehicle, the methods, apparatuses, and systems described herein can be applied to a variety of systems (e.g., a sensor system or a robotic platform), and is not limited to autonomous vehicles. In another example, the techniques can be utilized in an aviation or nautical context, or in any system involving agents or entity that may be associated with behavior that is unknown to the system. Further, although discussed in the context of LIDAR data, sensor data can include any two-dimensional, three-dimensional, or multi-dimensional data such as image data (e.g., stereo cameras, time-of-flight data, and the like)), RADAR data, SONAR data, and the like. Additionally, the techniques described herein can be used with real data (e.g., captured using sensor(s)), simulated data (e.g., generated by a simulator), or any combination of the two.

FIG. 1 is a pictorial flow diagram 100 of an example process for capturing sensor data, inputting images representing a top-down view of the environment into a prediction system, receiving heat maps associated with prediction probabilities, and generating a predicted trajectory based at least in part on the prediction probabilities, in accordance with embodiments of the disclosure.

At operation 102, the process can include capturing sensor data of an environment. In some examples, the sensor data can be captured by one or more sensors on an autonomous vehicle. For example, the sensor data can include data captured by one or more of a LIDAR sensor, a RADAR sensor, an image sensor, a time of flight sensor, a SONAR sensor, and the like.

An example 104 illustrates an environment in which an autonomous vehicle 106 is traversing the environment. In some examples, the autonomous vehicle 106 can perform the operations of the process 100, which can include capturing the sensor data of the environment. For the purpose of discussion, a vehicle capturing (or utilizing) the sensor data can be an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety critical functions for the entire trip, with the driver (or occupant) not being expected to control the vehicle at any time. In such an example, since the vehicle can be configured to control all functions from start to stop, including all parking functions, it can be unoccupied. This is merely an example, and the systems and methods described herein can be incorporated into any ground-borne, airborne, or waterborne vehicle, including those ranging from vehicles that need to be manually controlled by a driver at all times, to those that are partially or fully autonomously controlled. Additional details associated with the vehicle are described throughout this disclosure.

An agent 108 is also located in the environment. In some examples, the agent 108 can represent a vehicle, a pedestrian, a bicycle, an animal, and the like. The environment illustrated in the example 104 can include a driveable region 110 and a non-driveable region 112. In some examples, the operation 102 can include localizing the autonomous vehicle 106 in the environment and accessing map data associated with the environment. For example, map data can comprise map elements such as a crosswalk element 114 indicating a region of the environment corresponding to a crosswalk. Additional examples of map elements can include, but are not limited to, one or more of a lane element, a bike lane element, a crosswalk element, an intersection element, a lane divider element, a traffic light element, a stop sign element, a stop line element, a yield sign element, a yield line element, a parking lane element, a driveway element, a speed bump element, jay walking regions (e.g., a virtual crosswalk), trajectory waypoints (e.g., known trajectories), passenger pickup points, a sign location element, a geofence element, and the like.

In some instances, the sensor data captured in the operation 102 can be used to determine information about the agent 108, including but not limited to, a type of the agent (e.g., semantic information indicating a classification of the agent, such as a vehicle, pedestrian, bicycle, animal, and the like). In some instances, the operation 102 can include determining the type of the agent 108, a bounding box associated with the agent 108, and/or movement information associated with the agent, as discussed herein.

At operation 116, the process can include inputting image(s) representing the environment into a prediction system. In some instances, the images(s) can represent the environment from a top-down perspective, as illustrated in images 118, 120, and 122. That is, the images 118, 120, and 122 can represent the environment depicted in example 104 over time. For example, the image 118 can represent the environment at time T_(−N), the image 120 can represent the environment at time T_(−N+1), and the image 122 can represent the environment at time T₀. For example, the image 118 can represent the environment 2 seconds in the past, the image 120 can represent the environment 1 second in the past, and the image 122 can represent the environment at the present time (e.g., time=0). Of course, the images 118, 120, and 122 can represent the environment at any period in time, and are not limited to the express times discussed herein.

As introduced above, the image 118 can represent the environment at time T-N, and may include a bounding box 124 representing a location and extent (e.g., length and width) of the agent 108 at the indicated time. Further, the image 118 can include a bounding box 126 representing the autonomous vehicle 106. As can be understood, the image 118 can include any number of static objects (e.g., buildings, trees, curbs, sidewalks, map elements, etc.) or dynamic objects (e.g., agents) in the environment at any time, and is not limited to one agent (e.g., the bounding box 124 representing the agent 108), as discussed herein.

The image 118 can further include velocity information 128 which is represented in the image 118 as a velocity vector associated with the bounding box 124. As discussed above, the image 118 can include additional information representing objects in the environment or states of objects in the environment. For example, the image 118 can include lane information or corridor information indicating that a lane associated with the bounding box 126 (e.g., associated with the autonomous vehicle 106) is controlled by a traffic light, which, for the sake of simplicity, may correspond to a red light. Accordingly, in this example, a position of the bounding box 126 can remain constant in the images 118, 120, 122, although the autonomous vehicle 106 may be moving in the environment while capturing sensor data and determining at least one predicted trajectory, as discussed herein.

As can be understood, the bounding box 124 (e.g., the agent 108) can be associated with different locations and/or movement information over time. Accordingly, the image 120 can include velocity information 130 that is the same or different as the velocity information 128, and the image 122 can include velocity information 132 that is the same or different as the velocity information 128 and 130. Further, a location of the bounding box 124 is updated throughout the images 118, 120, and 122 to illustrate a movement of the agent 108 as the agent 108 traverses the environment.

Further, as discussed below in the context of FIGS. 4A and 4B, the image(s) input into the prediction system may be decomposed into a plurality of channels prior to input into the prediction system (e.g., FIG. 4A) or may be input into the prediction system without generating a plurality of channels (e.g., FIG. 4B).

At operation 134, the process can include receiving heat map(s) associated with prediction probabilities. In some examples, the operation 134 can include receiving heat maps 136, 138, and 140 from a prediction system, wherein the heat maps 136, 138, and 140 are based at least in part on the image(s) 118, 120, and 122 input into the prediction system in the operation 116. In some instances, the heat map(s) 136, 138, and 140 can represent (e.g., from a top-down perspective of the environment) the prediction probabilities indicative of a probability that the agent 108 will be at the corresponding location in the environment at the time corresponding to a respective heat map. That is, the heat maps 136, 138, and 140 can represent the prediction probabilities of the agent 108 in the future.

The heat map 136 can represent prediction probabilities 142 associated with the agent 108 at time T₁, the heat map 138 can represent prediction probabilities 144 associated with the agent 108 at time T₂, and the heat map 140 can represent prediction probabilities 146 associated with the agent 108 at time T_(M). For example, the heat map 136 can represent the prediction probabilities 142 at 1 second in the future, the heat map 138 can represent the prediction probabilities 144 at 2 seconds in the future, and the heat map 140 can represent the prediction probabilities 146 at the 5 seconds in the future. Of course, the heat maps 136, 138, and 140 can represent the prediction probabilities at any period in time, and are not limited to the express times discussed herein.

In some instances, the prediction probabilities 142, 144, and 146 can represent a probability distribution over the area represented by the respective heat maps 136, 138, and 140.

At operation 148, the operation can include determining predicted trajectory(ies) based at least in part on the prediction probabilities. In some instances, the operation 148 can include determining at least one predicted trajectory associated with the agent 108.

An example 150 illustrates predicted trajectories 152 and 154 associated with the agent 108, as determined by the autonomous vehicle 106. In some instances, the predicted trajectory 152 can represent a most probable predicted trajectory for the agent 108 to traverse in the environment based on the sensor data captured by the autonomous vehicle 106 and based on the outputs from the prediction system. In some instances, the predicted trajectory 154 can represent a second most probable predicted trajectory for the agent 108 to traverse in the environment, based on the aforementioned factors. That is, the operation 148 can include ranking or associating a confidence value or probability associated with each of the predicted trajectories to determine the most likely trajectory for the agent 108 to follow. In some instances, the predicted trajectories 152 and 154 can be associated with individual uncertainties. Additional details of predicted trajectories are discussed below in connection with FIGS. 3 and 5, as well as throughout this disclosure.

In some instances, the process 100 can be performed in parallel for each agent in the environment. In some instances, the process 100 can be performed on a single set of images to generate at least one trajectory for each agent of a plurality of agents in the environment.

FIG. 2 depicts a block diagram of an example system 200 for implementing the techniques described herein. In at least one example, the system 200 can include a vehicle 202.

The vehicle 202 can include a vehicle computing device 204, one or more sensor systems 206, one or more emitters 208, one or more communication connections 210, at least one direct connection 212, and one or more drive systems 214.

The vehicle computing device 204 can include one or more processors 216 and memory 218 communicatively coupled with the one or more processors 216. In the illustrated example, the vehicle 202 is an autonomous vehicle; however, the vehicle 202 could be any other type of vehicle. In the illustrated example, the memory 218 of the vehicle computing device 204 stores a localization component 220, a perception component 222, a planning component 224, one or more system controllers 226, one or more maps 228, and a prediction component 230 comprising an image generation component 232, a heat map generation component 234, a predicted point component 236, and a trajectory generation component 238. Though depicted in FIG. 2 as residing in memory 218 for illustrative purposes, it is contemplated that the localization component 220, the perception component 222, the planning component 224, the one or more system controllers 226, the one or more maps 228, the prediction component 230, the image generation component 232, the heat map generation component 234, the predicted point component 236, and the trajectory generation component 238 may additionally, or alternatively, be accessible to the vehicle 202 (e.g., stored remotely).

In at least one example, the localization component 220 can include functionality to receive data from the sensor system(s) 206 to determine a position and/or orientation of the vehicle 202 (e.g., one or more of an x-, y-, z-position, roll, pitch, or yaw). For example, the localization component 220 can include and/or request/receive a map of an environment and can continuously determine a location and/or orientation of the autonomous vehicle within the map. In some instances, the localization component 220 can utilize SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping, simultaneously), relative SLAM, bundle adjustment, non-linear least squares optimization, or the like to receive image data, LIDAR data, radar data, IMU data, GPS data, wheel encoder data, and the like to accurately determine a location of the autonomous vehicle. In some instances, the localization component 220 can provide data to various components of the vehicle 202 to determine an initial position of an autonomous vehicle for generating a trajectory and/or for generating map data, as discussed herein.

In some instances, the perception component 222 can include functionality to perform object detection, segmentation, and/or classification. In some examples, the perception component 222 can provide processed sensor data that indicates a presence of an entity that is proximate to the vehicle 202 and/or a classification of the entity as an entity type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 222 can provide processed sensor data that indicates one or more characteristics associated with a detected entity (e.g., a tracked object) and/or the environment in which the entity is positioned. In some examples, characteristics associated with an entity can include, but are not limited to, an x-position (global and/or local position), a y-position (global and/or local position), a z-position (global and/or local position), an orientation (e.g., a roll, pitch, yaw), an entity type (e.g., a classification), a velocity of the entity, an acceleration of the entity, an extent of the entity (size), etc. Characteristics associated with the environment can include, but are not limited to, a presence of another entity in the environment, a state of another entity in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc.

In general, the planning component 224 can determine a path for the vehicle 202 to follow to traverse through an environment. For example, the planning component 224 can determine various routes and trajectories and various levels of detail. For example, the planning component 224 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For the purpose of this discussion, a route can be a sequence of waypoints for travelling between two locations. As non-limiting examples, waypoints include streets, intersections, global positioning system (GPS) coordinates, etc. Further, the planning component 224 can generate an instruction for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 224 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instruction can be a trajectory, or a portion of a trajectory. In some examples, multiple trajectories can be substantially simultaneously generated (e.g., within technical tolerances) in accordance with a receding horizon technique, wherein one of the multiple trajectories is selected for the vehicle 202 to navigate.

In at least one example, the vehicle computing device 204 can include one or more system controllers 226, which can be configured to control steering, propulsion, braking, safety, emitters, communication, and other systems of the vehicle 202. These system controller(s) 226 can communicate with and/or control corresponding systems of the drive system(s) 214 and/or other components of the vehicle 202.

The memory 218 can further include one or more maps 228 that can be used by the vehicle 202 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV/HSL color information), and the like), intensity information (e.g., LIDAR information, RADAR information, and the like); spatial information (e.g., image data projected onto a mesh, individual “surfels” (e.g., polygons associated with individual color and/or intensity)), reflectivity information (e.g., specularity information, retroreflectivity information, BRDF information, BSSRDF information, and the like). In one example, a map can include a three-dimensional mesh of the environment. In some instances, the map can be stored in a tiled format, such that individual tiles of the map represent a discrete portion of an environment, and can be loaded into working memory as needed, as discussed herein. In at least one example, the one or more maps 228 can include at least one map (e.g., images and/or a mesh). In some examples, the vehicle 202 can be controlled based at least in part on the maps 228. That is, the maps 228 can be used in connection with the localization component 220, the perception component 222, and/or the planning component 224 to determine a location of the vehicle 202, identify objects in an environment, and/or generate routes and/or trajectories to navigate within an environment.

In some examples, the one or more maps 228 can be stored on a remote computing device(s) (such as the computing device(s) 242) accessible via network(s) 240. In some examples, multiple maps 228 can be stored based on, for example, a characteristic (e.g., type of entity, time of day, day of week, season of the year, etc.). Storing multiple maps 228 can have similar memory requirements, but increase the speed at which data in a map can be accessed.

In general, the prediction component 230 can generate predicted trajectories of objects in an environment. For example, the prediction component 230 can generate one or more predicted trajectories for vehicles, pedestrians, animals, and the like within a threshold distance from the vehicle 202. In some instances, the prediction component 230 can measure a trace of an object and generate a trajectory for the object based on observed and predicted behavior.

In some examples, the image generation component 232 can include functionality to generate images representing a top-down view of a scene. Examples of such images generated by the image generation component 232 are shown in FIG. 1 as images 118, 120, and 122.

In some examples, the image generation component 232 can generate an image representing an area (e.g., a receding horizon) around the vehicle 202. In some examples, the image may represent a 100 meter×100 meter area around the vehicle 202, although any area is contemplated. The image generation component 232 can receive data about agents in the environment from the perception component 222 and can receive data about the environment itself from the localization component 220, the perception component 222, and the one or more maps 228. The image generation component 232 can generate a top-down view of the environment including agents in the environment (e.g., represented by a bounding box, as discussed herein), semantic information about the agent (e.g., a classification type), movement information (e.g., velocity information, acceleration information, etc.), and the like. The image generation component 232 can further generate image elements (e.g., symbols) in the image representing the various attributes of the agent(s) and the environment, including but not limited to: one or more velocity vectors, one or more acceleration vectors, lighting indicators (e.g., braking lights of the agent, turn signal lights of the agent, etc., represented as respective indicator status), traffic light information (e.g., status of a traffic light), lane information (e.g., whether the lane is a left turn only lane, right turn only lane, etc.), and the like.

In some examples, the image generation component 232 can generate one or more images comprising semantic information of the environment, as discussed in detail herein.

In some examples, the image generation component 232 can generate one or more images representing a trace of an agent in the environment. As discussed above in FIG. 1, the images 118, 120, and 122 can represent prior information about the state of the environment and agents in the environment that can be used to generate at least one predicted trajectory, as discussed herein.

In any of the examples discussed herein, the image generation component 232 may generate an image having multiple channels, wherein each channel is representative of some information (semantic or otherwise).

In some examples, the heat map generation component 234, can include functionality to receive the image(s) generated by the image generation component 232 and to generate one or more heat maps including prediction probabilities, as discussed herein.

In some instances, the heat map generation component 234 can include one or more machine learning models trained to output one or more heat maps based on top-down images input into the model. In some examples, the heat map generation component 234 can include a convolutional neural network (CNN), which may or may not include one or more recurrent neural network (RNN) layers, e.g., long short-term memory (LSTM) layers.

In some examples, a heat map output by the heat map generation component 234 can represent a discretized region of the environment proximate to the autonomous vehicle. For example, the heat map can represent a 64×64 grid (or J×K sized grid) representing a 100 meter by 100 meter region around the autonomous vehicle. In some examples, an area represented by a heat map can correspond to the area represented by the images input to the heat map generation component 234. Of course, the heat map can represent any size of region and can represent any number of discrete portions of the region. In some cases, a portion of the heat map can be referred to as a cell of the heat map. Each cell can comprise a prediction probability representing a probability that the agent will be at the corresponding location in the environment at the time corresponding to the heat map. In some instances, the machine learning model can output a plurality of heat maps, wherein a heat map of the plurality of heat maps can represent probability predictions associated with the agent at a particular time in the future (e.g., 0.5 second, 1 second, 3 seconds, 5 seconds, 10 seconds, etc.).

In some examples, the heat map generation component 234 can be trained to generate any number of heat maps based on any number of input images. For example, the heat map generation component 234 can receive N number of input images and can output M number of heat maps.

In some examples, the predicted point component 236, can include functionality to determine predicted point(s) representing predicted location(s) of an agent in the environment. In some examples, the predicted point component 236 can receive at least one heat map from the heat map generation component 234. The predicted point component 236 can determine the predicted point associated with the heat map based at least in part on a cell associated with a highest probability and/or based at least in part on cost(s) associated with generating a predicted trajectory associated with the predicted point.

For example, the predicted point component 236 can select a point, cell, or region of a heat map as a predicted point based at least in part on evaluating one or more cost functions associated with vehicle dynamics. In at least some examples, such costs may be associated with preventing or reducing an occurrence where the predicted trajectory represents an unreasonable trajectory (e.g., involving “teleportation” of the vehicle, too high of a change in velocity, etc.). Examples of such costs may include, but are not limited to, a positional based cost (e.g., limiting the distance allowed between predicted points), a velocity cost (e.g., a constant velocity cost enforcing a constant velocity through the predicted trajectory), an acceleration cost (e.g., enforcing acceleration bounds throughout the predicted trajectory), an expectation that the agent may follow rules of the road, and the like. In at least some examples, the probability associated with the cell may be multiplied with the cost (which, in at least some examples, may be normalized) such that the point (e.g., a candidate point) associated with the highest value of the cost times probability is selected as the predicted point.

Further, in some cases, the predicted point component 236 can determine predicted points for a plurality of heat maps and can provide the predicted points to the trajectory generation component 238 to determine a predicted trajectory based at least in part on the predicted points.

In some instances, after determining a predicted point for each heat map, the predicted point component 236 can receive a predicted trajectory from the trajectory generation component 238 and can mask regions of the heat map associated with the predicted trajectory to generate a masked heat map. The predicted point component 236 can normalize the remaining, unmasked prediction probabilities and can determine a predicted point associated with the masked heat map. An additional predicted point can determined for each masked heat map, and the additional predicted points can be provided to the trajectory generation component 238 to determine an additional predicted trajectory. In some instances, the masking operations and the operations of determining predicted points can be repeated until a prediction probability or a probability of a resulting predicted trajectory is below a threshold value. In other examples, such a process can be repeated until the heat map is completely masked. Additional details of determining a predicted point are discussed below in connection with FIG. 3.

In some instances, the predicted point component 236 can receive images generated by the image generation component 232 to generate one or more predicted points directly (e.g., without determining the predicted points from heat map(s) provided by the heat map generation component 234). In such a case, the predicted point component 236 can comprise a CNN, which may or may not include one or more RNN (which may be LSTM) layers.

In some examples, the trajectory generation component 238 can include functionality to receive one or more predicted points from the predicted point component 236 to generate one or more predicted trajectories. In some examples, the trajectory generation component 238 can generate a curve based at least in part on the predicted points as a predicted trajectory.

In some instances, the trajectory generation component 238 can provide the at least one predicted trajectory to the planning component 224, which in turn can generate a trajectory for the vehicle 202 to follow.

In some instances, the heat map generation component 234 can provide the heat map(s) directly to the planning component 224 (e.g., with or without predicted trajectories), which in turn can generate a trajectory for the vehicle 202 to follow.

In some instances, aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms. For example, in some instances, the components in the memory 218 (and the memory 246, discussed below) can be implemented as a neural network.

As described herein, an exemplary neural network is a biologically inspired algorithm which passes input data through a series of connected layers to produce an output. Each layer in a neural network can also comprise another neural network, or can comprise any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network can utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based on learned parameters.

Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decisions tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), Chi-squared automatic interaction detection (CHAID), decision stump, conditional decision trees), Bayesian algorithms (e.g., naïve Bayes, Gaussian naïve Bayes, multinomial naïve Bayes, average one-dependence estimators (AODE), Bayesian belief network (BNN), Bayesian networks), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, back-propagation, hopfield network, Radial Basis Function Network (RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Network (CNN), Stacked Auto-Encoders), Dimensionality Reduction Algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc.

Additional examples of architectures include neural networks such as ResNet70, ResNet101, VGG, DenseNet, PointNet, and the like.

In at least one example, the sensor system(s) 206 can include LIDAR sensors, RADAR sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), time of flight sensors, microphones, wheel encoders, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor system(s) 206 can include multiple instances of each of these or other types of sensors. For instance, the LIDAR sensors can include individual LIDAR sensors located at the corners, front, back, sides, and/or top of the vehicle 202. As another example, the camera sensors can include multiple cameras disposed at various locations about the exterior and/or interior of the vehicle 202. The sensor system(s) 206 can provide input to the vehicle computing device 204. Additionally or alternatively, the sensor system(s) 206 can send sensor data, via the one or more networks 240, to the one or more computing device(s) at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc.

The vehicle 202 can also include one or more emitters 208 for emitting light and/or sound, as described above. The emitters 208 in this example include interior audio and visual emitters to communicate with passengers of the vehicle 202. By way of example and not limitation, interior emitters can include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitters 208 in this example also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include lights to signal a direction of travel or other indicator of vehicle action (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) to audibly communicate with pedestrians or other nearby vehicles, one or more of which comprising acoustic beam steering technology.

The vehicle 202 can also include one or more communication connection(s) 210 that enable communication between the vehicle 202 and one or more other local or remote computing device(s). For instance, the communication connection(s) 210 can facilitate communication with other local computing device(s) on the vehicle 202 and/or the drive system(s) 214. Also, the communication connection(s) 210 can allow the vehicle to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). The communications connection(s) 210 also enable the vehicle 202 to communicate with a remote teleoperations computing device or other remote services.

The communications connection(s) 210 can include physical and/or logical interfaces for connecting the vehicle computing device 204 to another computing device or a network, such as network(s) 240. For example, the communications connection(s) 210 can enable Wi-Fi-based communication such as via frequencies defined by the IEEE 802.11 standards, short range wireless frequencies such as Bluetooth, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.) or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).

In at least one example, the vehicle 202 can include one or more drive systems 214. In some examples, the vehicle 202 can have a single drive system 214. In at least one example, if the vehicle 202 has multiple drive systems 214, individual drive systems 214 can be positioned on opposite ends of the vehicle 202 (e.g., the front and the rear, etc.). In at least one example, the drive system(s) 214 can include one or more sensor systems to detect conditions of the drive system(s) 214 and/or the surroundings of the vehicle 202. By way of example and not limitation, the sensor system(s) can include one or more wheel encoders (e.g., rotary encoders) to sense rotation of the wheels of the drive systems, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors to acoustically detect objects in the surroundings of the drive system, LIDAR sensors, RADAR sensors, etc. Some sensors, such as the wheel encoders can be unique to the drive system(s) 214. In some cases, the sensor system(s) on the drive system(s) 214 can overlap or supplement corresponding systems of the vehicle 202 (e.g., sensor system(s) 206).

The drive system(s) 214 can include many of the vehicle systems, including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and/or pneumatic components, a stability control system for distributing brake forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head/tail lights to illuminate an exterior surrounding of the vehicle), and one or more other systems (e.g., cooling system, safety systems, onboard charging system, other electrical components such as a DC/DC converter, a high voltage j unction, a high voltage cable, charging system, charge port, etc.). Additionally, the drive system(s) 214 can include a drive system controller which can receive and preprocess data from the sensor system(s) and to control operation of the various vehicle systems. In some examples, the drive system controller can include one or more processors and memory communicatively coupled with the one or more processors. The memory can store one or more components to perform various functionalities of the drive system(s) 214. Furthermore, the drive system(s) 214 also include one or more communication connection(s) that enable communication by the respective drive system with one or more other local or remote computing device(s).

In at least one example, the direct connection 212 can provide a physical interface to couple the one or more drive system(s) 214 with the body of the vehicle 202. For example, the direct connection 212 can allow the transfer of energy, fluids, air, data, etc. between the drive system(s) 214 and the vehicle. In some instances, the direct connection 212 can further releasably secure the drive system(s) 214 to the body of the vehicle 202.

In some examples, the vehicle 202 can send sensor data to one or more computing device(s) 242 via the network(s) 240. In some examples, the vehicle 202 can send raw sensor data to the computing device(s) 242. In other examples, the vehicle 202 can send processed sensor data and/or representations of sensor data to the computing device(s) 242. In some examples, the vehicle 202 can send sensor data to the computing device(s) 242 at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc. In some cases, the vehicle 202 can send sensor data (raw or processed) to the computing device(s) 242 as one or more log files.

The computing device(s) 242 can include processor(s) 244 and a memory 246 storing a training component 248 and a prediction component 250.

In some instances, the training component 248 can include functionality to train a machine learning model to output one or more predicted points, one or more confidence values associated with the one or more predicted points, or one or more heat maps including prediction probabilities. For example, the training component 248 can receive a set of images representing a top-down view of an environment. In some cases, the set of images can represent an agent traversing through an environment for a period of time, such as 5 seconds, 7 seconds, 10 seconds, 20 seconds, and the like. At least a portion of the set of images can be used as an input to train the machine learning model, and at least a portion of the set of images can be used as ground truth information for training the machine learning model. As a non-limiting example, a first set (e.g., 3, 4, 5, or more) of a sequence of images may be input into the machine learned model. A second set of images (or trajectory information associated therefrom—e.g., by extracting positions and/or velocities from the images) in the sequence of images immediately preceding the first set may then be used as ground truth for training the model. Thus, by providing images of recorded trajectories where agents traverse an environment, the prediction components 230 and 250 can be trained to output predicted points, confidence values associated with the predicted points, or one or more heat maps including prediction probabilities, as discussed herein.

In some instances, the prediction component 250 can substantially correspond to the prediction component 230 and can include functionality to generate predicted trajectories of objects in an environment. For example, the prediction component 250 can generate one or more predicted trajectories for vehicles, pedestrians, animals, and the like within a threshold distance from the vehicle 202. In some instances, the prediction component 250 can measure a trace of an object and generate a trajectory for the object based on observed and predicted behavior.

The processor(s) 216 of the vehicle 202 and the processor(s) 244 of the computing device(s) 242 can be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) 216 and 244 can comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that can be stored in registers and/or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices can also be considered processors in so far as they are configured to implement encoded instructions.

Memory 218 and 246 are examples of non-transitory computer-readable media. The memory 218 and 246 can store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems. In various implementations, the memory can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples that are related to the discussion herein.

In some instances, the memory 218 and 246 can include at least a working memory and a storage memory. For example, the working memory may be a high-speed memory of limited capacity (e.g., cache memory) that is used for storing data to be operated on by the processor(s) 216 and 244. In some instances, the memory 218 and 246 can include a storage memory that may be a lower-speed memory of relatively large capacity that is used for long-term storage of data. In some cases, the processor(s) 216 and 244 cannot operate directly on data that is stored in the storage memory, and data may need to be loaded into a working memory for performing operations based on the data, as discussed herein.

It should be noted that while FIG. 2 is illustrated as a distributed system, in alternative examples, components of the vehicle 202 can be associated with the computing device(s) 242 and/or components of the computing device(s) 242 can be associated with the vehicle 202. That is, the vehicle 202 can perform one or more of the functions associated with the computing device(s) 242, and vice versa.

FIG. 3 is a pictorial flow diagram of an example process 300 for receiving heat maps associated with prediction probabilities, determining a predicted point based at least in part on a highest probability in each heat map, and generating one or more predicted trajectories based at least in part on a plurality of predicted points, in accordance with embodiments of the disclosure.

At operation 302, the process can include receiving heat map(s) associated with prediction probabilities. In some instances, the operation 302 can include receiving the heat maps 136, 138, and 140, as discussed above in connection with FIG. 1. The heat map 136 can include the prediction probabilities 142, which may represent a probability of a location of the agent 108 in the environment at time T₁. The heat map 138 can include the prediction probabilities 144, which may represent a probability of a location of the agent 108 in the environment at time T₂. And the heat map 140 can include the prediction probabilities 146, which may represent a probability of a location of the agent 108 in the environment at time T_(M).

At operation 304, the process can include determining predicted point(s) based at least in part on a highest probability and on cost(s). FIG. 3 illustrates examples 306, 308, and 310 representing a predicted point as determined from the heat maps 136, 138, and 140, respectively.

The example 306 illustrates the predicted point 312 representing a highest probability of the prediction probabilities 142 included in the heat map 136. In some instances, for a first heat map in a series of heat maps, the first predicted point (e.g., the predicted point 312) can correspond to a highest probability of the heat map 136.

The example 308 illustrates the predicted point 314 based at least in part on a highest probability of the prediction probabilities 144 included in the heat map 138 and on one or more costs. For example, the predicted point 314 can be selected to maximize a probability associated with the predicted point 314 and to penalize selecting a point (e.g., a candidate point) as a predicted point based on one or more cost functions associated with vehicle dynamics to prevent or reduce an occurrence where the predicted trajectory represents an unreasonable trajectory (e.g., involving “teleportation” of the vehicle), or otherwise has a high associated cost. Examples of such costs and/or constraints may include, but are not limited to, a positional cost (e.g., such that subsequent predicted points selected follow a physically plausible trajectory), a velocity cost (e.g., a constant velocity cost), an acceleration cost, an expectation that the agent may follow rules of the road, and the like.

The example 310 illustrates a candidate point 316 and a predicted point 318. In some examples, the candidate point 316 may represent a highest probability of the prediction probabilities 146 included in the heat map 140. In some examples, the predicted point 318 may represent a probability that is less than the highest probability as the candidate point 316. However, because selecting the candidate point 316 as the predicted point would incur high cost because such a trajectory based on the predicted points 312 and 314 and the candidate point 316 would represent the agent 108 “teleporting” from a location associated with the predicted point 314 to a location associated with the candidate point 316.

Instead, the operation 304 can include selecting the predicted point 318 based at least in part on costs associated with a predicted trajectory based at least in part on the predicted points 312, 314, and 318.

At operation 320, the process can include determining a predicted trajectory based one the predicted points. As illustrated in an example 322, the operation 320 can interpolating between the predicted points 312, 314, and 318 to determine a predicted trajectory 324.

At operation 326, the process can include masking region(s) associated with a predicted trajectory. Examples 328, 330, and 332 illustrate the heat maps 136, 138, and 140, respectively, with masks 334, 336, and 338 obscuring (in whole or in part) the prediction probabilities 142, 144, and 146, respectively. In some examples, the mask 334 can represent a region associated with the predicted point 312. For example, the mask 334 can represent a circle or other polygon center around or associated with the predicted point 312. In some examples, the mask 334 can represent a circle having a radius of 1 meter, although the mask 334 can represent any size. In some examples, the mask 336 can be associated with the predicted point 314, and the mask 338 can be associated with the predicted point 318.

In some examples, a mask can be based at least in part on the predicted trajectory 324. For example, a mask can correspond to a lane element represented in map data traversed by the predicted trajectory 324.

In some examples, the operation 326 can generate masked heat maps corresponding to the heat maps 136, 138, and 140, respectively.

At operation 340, the process can include determining whether prediction probabilities are under a threshold value. For example, the operation 340 can include determining whether the unmasked portions of the prediction probabilities associated with each heat map are under a threshold value. If “no,” (indicating there are prediction probabilities that meet or exceed the threshold value), the process can return to the operation 304 to determine one or more predicted points based on the masked heat maps, which in turn can be used to determine additional predicted trajectories.

For example, because the candidate point 316 was not masked by the mask 338 in the example 332, such a predicted point could be used to determine an additional predicted trajectory.

In some examples, the operation 340 or 304 can include normalizing the unmasked prediction probabilities (e.g., scaling the unmasked prediction probabilities between 0 and 1) prior to determining an additional predicted trajectory.

When there are no prediction probabilities under the threshold (e.g., “yes” in the operation 340), the process can continue to operation 342. In at least some examples, operation may continue to operation 304 if there are any unmasked portions of the heatmap remaining (or less than some threshold amount). In such examples, all possible (or probable) trajectories may be determined in accordance with the techniques described herein.

At operation 342, the process can include outputting predicted trajectory(ies) to a planning system to generate trajectory(ies) to control an autonomous vehicle. An example 344 illustrates the predicted trajectory 324 (e.g., based at least in part on the predicted points 312, 314, and 318) and a predicted trajectory 346 (e.g., based at least in part on the candidate point 316, which may have been determined to be a predicted point for the predicted trajectory 346). In some examples, the predicted trajectories can be output to the planning component 224 of the autonomous vehicle 106. The planning system 224 can generate one or more trajectories for the autonomous vehicle 106 to follow based at least in part on the at least one predicted trajectory. In at least some examples, the trajectory(ies) may comprise uncertainties associated with each predicted point, an uncertainty of the trajectory as a whole, or a heat map.

FIGS. 4A and 4B are example illustrations of inputs to the prediction system to determine one or more predicted trajectories, in accordance with embodiments of the disclosure.

FIG. 4A illustrates an example 400 where an input image 402 is used to generate a plurality of channels 404, which are subsequently input to the prediction component 230 to determine one or more predicted trajector(ies) 406

In some examples, individual aspects of the input image 402 can be used to generate an individual channel of the channels 404. For example, an individual channel may represent, but are not limited to, one or more of: a bounding box (e.g., the bounding box 124, with a color of the bounding box representing a classification of the agent associated with the bounding box), a velocity of the bounding box in an x-direction and a y-direction (where a velocity can be represented as a vector in one channel or in multiple channels, with the x-velocity and the y-velocity represented in different channels), an acceleration of the agent in an x-direction and a y-direction (where an acceleration can be represented as a vector in one channel or in multiple channels, with the x-acceleration and the y-acceleration represented in different channels), a blinker status associated with the bounding box (e.g., left-turn, right-turn, braking, reverse, hazards, no lights, etc.), road network information (e.g., semantic labels indicating lanes, crosswalks, stop signs, intersections, traffic lights, and the like), traffic light status (e.g., red light, yellow light, green light, etc.), and the like. In some examples, the plurality of channels can be input to the prediction system to generate at least one predicted trajectory.

FIG. 4B illustrates an example 408 where the input image 402 is input to the prediction component 230 to determine the predicted trajectories 406. In contrast to the example 400, the example 408 may not include generating a plurality of channels associated with the image 402, and may instead include directly inputting the image 402 into the prediction component 230.

FIG. 5 is an illustration 500 of various predicted output(s) 502 by a prediction system, in accordance with embodiments of the disclosure. The illustration 500 provides three representative examples 504, 506, and 508 of the predicted output(s) 502 output by the prediction component 230, as discussed herein.

The example 504 shows predicted points 510 and an associated predicted trajectory 512. In some instances, the predicted points 510 can be output directly by the predicted point component 236 of FIG. 2. In some cases, the predicted trajectory 512 can be generated by interpolating between the predicted points 510 and may include smoothing the predicted trajectory 512 to optimize vehicle dynamics, comfort, and the like.

The example 504 includes an individual predicted point 514 associated with individual uncertainty information 516. In some examples, the uncertainty information 516 can represent a region of the environment indicative of a particular probability that a location of the agent 108 will be located with the area represented by the uncertainty information 516 at the time associated with the predicted point 514. In some examples, the certainty information can be output by the predicted point component 236 in association with the predicted point 514.

The example 506 shows another embodiment of predicted trajectories. In some examples, the predicted trajectory 512 and a predicted trajectory 518 can be output by the prediction component 230 in accordance with the techniques discussed in FIG. 3, as well as throughout this disclosure. That is, the example 506 can represent the predicted trajectories based at least in part on one or more heat maps, which can include multiple trajectories with probabilities, rankings, or weights associated with individual trajectories. In some instances, the individual predicted trajectories 512 and 518 can be associated with individual uncertainties.

The example 508 illustrates “raw” heat maps 136, 138 and 140 that can be output by the prediction component 230 for use in the planning component 224. That is, in some instances, the prediction component 230 can output the heat maps 136, 138, and 140 in addition to or instead of outputs illustrated by the examples 504 and 506.

Thus, the illustration 500 demonstrates the flexible outputs of the prediction component 230 discussed herein, whereby an output format of the predicted output(s) 502 is based on a particular implementation of the system.

FIGS. 1, 3, and 6 illustrate example processes in accordance with embodiments of the disclosure. These processes are illustrated as logical flow graphs, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.

FIG. 6 depicts an example process 600 for generating predicted trajectories based at least in part on a top-down representation of an environment including information associated with an agent in the environment, in accordance with embodiments of the disclosure. For example, some or all of the process 600 can be performed by one or more components in FIG. 2, as described herein. For example, some or all of the process 600 can be performed by the vehicle computing device(s) 204 and/or the computing device(s) 242.

At operation 602, the process can include capturing sensor data of an environment using a sensor of an autonomous vehicle. For example, the sensor data can include data captured by one or more of a LIDAR sensor, a RADAR sensor, an image sensor, a time of flight sensor, a SONAR sensor, and the like. In some examples, the operation 602 can include capturing sensor data using a plurality of sensors and fusing or combining the sensor data into a detail and informative representation of the environment.

At operation 604, the process can include determining whether an agent is represented in the sensor data. For example, the operation 604 can include inputting at least a portion of the sensor data into a detector or providing the data to a detection or perception algorithm (e.g., the perception component 222) to identify one or more agents in the environment. Examples of agents include, but are not limited to, a vehicle, a pedestrian, a bicycle, a skateboard, a scooter, an animal, and the like. If no agent is detected (e.g., “no” in the operation 604) the process can return to the operation 602 to captured additional sensor data. If an agent is detected (e.g., “yes” in the operation 604

At operation 606, the process can include generating, based at least in part on the sensor data, an image representing a top-down view of the environment, the image comprising a bounding box associated with the agent and velocity information associated with the agent. In some examples, the image can comprise acceleration information (instead of or in addition to the velocity information).

Additional information that can be included in the image can include, but is not limited to: a color of the bounding box representing a classification of the agent associated with the bounding box, a velocity of the bounding box in an x-direction and a y-direction (where a velocity can be represented as a vector in one channel or in multiple channels, with the x-velocity and the y-velocity represented in different channels), an acceleration of the agent in an x-direction and a y-direction (where an acceleration can be represented as a vector in one channel or in multiple channels, with the x-acceleration and the y-acceleration represented in different channels), a blinker status associated with the bounding box (e.g., left-turn, right-turn, braking, reverse, hazards, no lights, etc.), road network information (e.g., semantic labels indicating lanes, crosswalks, stop signs, intersections, traffic lights, and the like), traffic light status (e.g., red light, yellow light, green light, etc.), and the like. In such an image, semantic information about the agent and the environment may be encoded in one or more channels as discussed in detail herein.

As discussed herein, the operation 606 can include generating a plurality of images representing behavior of the agent in the environment over time.

At operation 608, the process can include inputting the image into a machine learning model trained to generate a heat map associated with prediction probabilities. In some examples, the operation 608 can include generating a plurality of channels based on the image and inputting the plurality of channels into the machine learning model. In some examples, the image can be input into the machine learning model without generating a plurality of channels. Additional details of the machine learning model, the heat map, and the prediction probabilities are discussed throughout the disclosure.

At operation 610, the process can include determining, based at least in part on the heat map, a predicted trajectory associated with the agent. Additional details of determining a predicted trajectory based at least in part on the heat map are discussed above in connection with FIG. 3, as well as throughout the disclosure.

At operation 612, the process can include determining, based at least in part on the predicted trajectory, a trajectory for the autonomous vehicle. In some examples, the trajectory based on the predicted trajectory can represent a safer and/or smoother trajectory compared to a trajectory generated without the predicted trajectory, as the planning component 224 generating the trajectory can anticipate more closely the actions of entities proximate to the autonomous vehicle.

At operation 614, the process can include controlling the autonomous vehicle based at least in part on the trajectory. In some examples, the operation 614 can include controlling one or more of steering, braking, and/or acceleration of the autonomous vehicle to traverse through the environment.

Example Clauses

A: A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: capturing sensor data of an environment using at least one sensor of an autonomous vehicle; generating, based at least in part on the sensor data, an image representing a top-down view of the environment, the image comprising a bounding box associated with a vehicle and velocity information associated with the vehicle, the vehicle being separate from the autonomous vehicle; inputting the image into a machine learning model trained to generate a heat map associated with prediction probabilities of possible locations associated with the vehicle; determining, based at least in part on the heat map, a predicted trajectory associated with the vehicle; and determining, based at least in part on the predicted trajectory, a trajectory for the autonomous vehicle to travel in the environment.

B: The system of paragraph A, wherein determining the predicted trajectory comprises: determining a predicted point based at least in part on a highest probability portion associated with a portion of the heat map and a cost associated with determining the predicted trajectory based at least in part on the predicted point.

C: The system of paragraph B, wherein: the predicted point is a first predicted point; the highest probability portion is a first highest probability portion; the portion is a first portion; the cost is a first cost; the predicted trajectory is a first predicted trajectory; and the operations further comprise: masking, as a masked heat map, a second portion of the heat map associated with the first predicted point, wherein the second portion of the heat map comprises at least the first portion; determining, as a second predicted point, a second highest probability portion associated with a third portion of the masked heat map; and determining a second predicted trajectory based at least in part on the second predicted point and on a second cost associated with determining the second predicted trajectory based at least in part on the second predicted point.

D: The system of any of paragraphs A-C, wherein the image comprises: a first channel representing the bounding box and semantic information associated with the vehicle; and a second channel representing the velocity information associated with the vehicle, and wherein the operations further comprise inputting the first channel and the second channel into the machine learning model.

E: The system of any of paragraphs A-D, wherein the machine learning model comprises: a convolutional neural network (CNN); and a long short-term memory (LSTM) component coupled to the CNN.

F: A method comprising: receiving sensor data of an environment; generating an image representing a top-down view of an agent in the environment, wherein the image includes semantic information associated with the agent or the environment; inputting the image into an algorithm configured to generate a probability distribution associated with a predicted location of the agent; determining, based at least in part on the probability distribution, a predicted trajectory associated with the agent; and controlling an autonomous vehicle based at least in part on the predicted trajectory.

G: The method of paragraph F, wherein the probability distribution is represented as a heat map discretizing portions of the environment, wherein a prediction probability of the probability distribution is association with a portion of the heat map.

H: The method of paragraph G, wherein determining the predicted trajectory comprises: determining a predicted point based at least in part on a highest probability portion of the heat map and a cost associated with the predicted trajectory based at least in part on the predicted point.

I: The method of paragraph H, wherein: the predicted point is a first predicted point; the highest probability portion is a first highest probability portion; the portion is a first portion; the cost is a first cost; the predicted trajectory is a first predicted trajectory; and the method further comprises: masking, as a masked heat map, a second portion of the heat map associated with the first predicted point, wherein the second portion of the heat map comprises at least the first portion; and determining a second predicted point based at least in part on a second highest probability portion of the heat map and a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point.

J: The method of any of paragraphs F-I, wherein the image comprises a plurality of channels, a channel of the plurality of channels comprising at least one of: a location of the agent; an extent of the agent; a bounding box associated with the agent; velocity information associated with the agent; acceleration information associated with the agent; turn indicator status of the agent; map data associated with the environment; or traffic light information associated with the environment.

K: The method of any of paragraphs F-J, wherein: the algorithm is a machine learning model comprising a convolutional neural network trained to generate a heat map representing the probability distribution; and the semantic information comprises a classification of the agent and velocity information associated with the agent.

L: The method of any of paragraphs F-K, wherein controlling the autonomous vehicle based at least in part on the predicted trajectory comprises: generating a trajectory for the autonomous vehicle to follow to traverse the environment; and controlling the autonomous vehicle based at least in part on the trajectory.

M: The method of any of paragraphs F-L, further comprising: capturing, using a sensor on the autonomous vehicle, the sensor data of the environment, wherein the sensor comprises one or more of: a LIDAR sensor; a RADAR sensor; an image sensor; or a time of flight sensor.

N: The method of any of paragraphs F-M, wherein the predicted trajectory is associated with an uncertainty.

O: The method of any of paragraphs F-N, wherein: the sensor data represents the environment at a first time; the probability distribution is a first probability distribution associated with a second time after the first time; the predicted location is a first predicted location; and the algorithm is further configured to generate: a second probability distribution associated with a second predicted location of the agent at a third time after the second time.

P: A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising: receiving sensor data of an environment; generating an image representing a top-down view of an agent in the environment, wherein the image includes semantic information associated with the agent or the environment; inputting the image into an algorithm configured to generate a probability distribution associated with a predicted location of the agent; determining, based at least in part on the probability distribution, a predicted trajectory associated with the agent; and controlling an autonomous vehicle based at least in part on the predicted trajectory.

Q: The non-transitory computer-readable medium of paragraph P, wherein the probability distribution is represented as a heat map discretizing portions of the environment, wherein a prediction probability of the probability distribution is association with a portion of the heat map.

R: The non-transitory computer-readable medium of paragraph Q, wherein determining the predicted trajectory comprises: determining a predicted point based at least in part on a highest probability portion associated with the heat map and a cost.

S: The non-transitory computer-readable medium of paragraph R, wherein: the predicted point is a first predicted point; the highest probability portion is a first highest probability portion; the portion is a first portion; the cost is a first cost; the predicted trajectory is a first predicted trajectory; and the operations further comprise: masking, as a masked heat map, a second portion of the heat map associated with the first predicted point, wherein the second portion of the heat map comprises at least the first portion; and determining a second predicted point based at least in part on a second highest probability portion associated with a third portion of the masked heat map and on a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point.

T: The non-transitory computer-readable medium of any of paragraphs P-S, wherein: the sensor data represents the environment at a first time; the probability distribution is a first probability distribution associated with a second time after the first time; and the probability distribution is generated prior to the second time.

While the example clauses described above are described with respect to one particular implementation, it should be understood that, in the context of this document, the content of the example clauses can also be implemented via a method, device, system, a computer-readable medium, and/or another implementation.

CONCLUSION

While one or more examples of the techniques described herein have been described, various alterations, additions, permutations and equivalents thereof are included within the scope of the techniques described herein.

In the description of examples, reference is made to the accompanying drawings that form a part hereof, which show by way of illustration specific examples of the claimed subject matter. It is to be understood that other examples can be used and that changes or alterations, such as structural changes, can be made. Such examples, changes or alterations are not necessarily departures from the scope with respect to the intended claimed subject matter. While the steps herein can be presented in a certain order, in some cases the ordering can be changed so that certain inputs are provided at different times or in a different order without changing the function of the systems and methods described. The disclosed procedures could also be executed in different orders. Additionally, various computations that are herein need not be performed in the order disclosed, and other examples using alternative orderings of the computations could be readily implemented. In addition to being reordered, the computations could also be decomposed into sub-computations with the same results. 

What is claimed is:
 1. A system comprising: one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising: capturing sensor data of an environment using at least one sensor of an autonomous vehicle; generating, based at least in part on the sensor data, an image representing a top-down view of the environment, the image comprising a bounding box associated with a vehicle and velocity information associated with the vehicle, the vehicle being separate from the autonomous vehicle; inputting the image into a machine learning model trained to generate a heat map associated with prediction probabilities of possible locations associated with the vehicle; determining, based at least in part on the heat map, a predicted trajectory associated with the vehicle; and determining, based at least in part on the predicted trajectory, a trajectory for the autonomous vehicle to travel in the environment.
 2. The system of claim 1, wherein determining the predicted trajectory comprises: determining a predicted point based at least in part on a highest probability portion associated with a portion of the heat map and a cost associated with determining the predicted trajectory based at least in part on the predicted point.
 3. The system of claim 2, wherein: the predicted point is a first predicted point; the highest probability portion is a first highest probability portion; the portion is a first portion; the cost is a first cost; the predicted trajectory is a first predicted trajectory; and the operations further comprise: masking, as a masked heat map, a second portion of the heat map associated with the first predicted point, wherein the second portion of the heat map comprises at least the first portion; determining, as a second predicted point, a second highest probability portion associated with a third portion of the masked heat map; and determining a second predicted trajectory based at least in part on the second predicted point and on a second cost associated with determining the second predicted trajectory based at least in part on the second predicted point.
 4. The system of claim 1, wherein the image comprises: a first channel representing the bounding box and semantic information associated with the vehicle; and a second channel representing the velocity information associated with the vehicle, and wherein the operations further comprise inputting the first channel and the second channel into the machine learning model.
 5. The system of claim 1, wherein the machine learning model comprises: a convolutional neural network (CNN); and a long short-term memory (LSTM) component coupled to the CNN.
 6. A method comprising: receiving sensor data of an environment; generating an image representing a top-down view of an agent in the environment, wherein the image includes semantic information associated with the agent or the environment; inputting the image into an algorithm configured to generate a probability distribution associated with a predicted location of the agent; determining, based at least in part on the probability distribution, a predicted trajectory associated with the agent; and controlling an autonomous vehicle based at least in part on the predicted trajectory.
 7. The method of claim 6, wherein the probability distribution is represented as a heat map discretizing portions of the environment, wherein a prediction probability of the probability distribution is association with a portion of the heat map.
 8. The method of claim 7, wherein determining the predicted trajectory comprises: determining a predicted point based at least in part on a highest probability portion of the heat map and a cost associated with the predicted trajectory based at least in part on the predicted point.
 9. The method of claim 8, wherein: the predicted point is a first predicted point; the highest probability portion is a first highest probability portion; the portion is a first portion; the cost is a first cost; the predicted trajectory is a first predicted trajectory; and the method further comprises: masking, as a masked heat map, a second portion of the heat map associated with the first predicted point, wherein the second portion of the heat map comprises at least the first portion; and determining a second predicted point based at least in part on a second highest probability portion of the heat map and a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point.
 10. The method of claim 6, wherein the image comprises a plurality of channels, a channel of the plurality of channels comprising at least one of: a location of the agent; an extent of the agent; a bounding box associated with the agent; velocity information associated with the agent; acceleration information associated with the agent; turn indicator status of the agent; map data associated with the environment; or traffic light information associated with the environment.
 11. The method of claim 6, wherein: the algorithm is a machine learning model comprising a convolutional neural network trained to generate a heat map representing the probability distribution; and the semantic information comprises a classification of the agent and velocity information associated with the agent.
 12. The method of claim 6, wherein controlling the autonomous vehicle based at least in part on the predicted trajectory comprises: generating a trajectory for the autonomous vehicle to follow to traverse the environment; and controlling the autonomous vehicle based at least in part on the trajectory.
 13. The method of claim 6, further comprising: capturing, using a sensor on the autonomous vehicle, the sensor data of the environment, wherein the sensor comprises one or more of: a LIDAR sensor; a RADAR sensor; an image sensor; or a time of flight sensor.
 14. The method of claim 6, wherein the predicted trajectory is associated with an uncertainty.
 15. The method of claim 6, wherein: the sensor data represents the environment at a first time; the probability distribution is a first probability distribution associated with a second time after the first time; the predicted location is a first predicted location; and the algorithm is further configured to generate: a second probability distribution associated with a second predicted location of the agent at a third time after the second time.
 16. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising: receiving sensor data of an environment; generating an image representing a top-down view of an agent in the environment, wherein the image includes semantic information associated with the agent or the environment; inputting the image into an algorithm configured to generate a probability distribution associated with a predicted location of the agent; determining, based at least in part on the probability distribution, a predicted trajectory associated with the agent; and controlling an autonomous vehicle based at least in part on the predicted trajectory.
 17. The non-transitory computer-readable medium of claim 16, wherein the probability distribution is represented as a heat map discretizing portions of the environment, wherein a prediction probability of the probability distribution is association with a portion of the heat map.
 18. The non-transitory computer-readable medium of claim 17, wherein determining the predicted trajectory comprises: determining a predicted point based at least in part on a highest probability portion associated with the heat map and a cost.
 19. The non-transitory computer-readable medium of claim 18, wherein: the predicted point is a first predicted point; the highest probability portion is a first highest probability portion; the portion is a first portion; the cost is a first cost; the predicted trajectory is a first predicted trajectory; and the operations further comprise: masking, as a masked heat map, a second portion of the heat map associated with the first predicted point, wherein the second portion of the heat map comprises at least the first portion; and determining a second predicted point based at least in part on a second highest probability portion associated with a third portion of the masked heat map and on a second cost associated with determining a second predicted trajectory based at least in part on the second predicted point.
 20. The non-transitory computer-readable medium of claim 16, wherein: the sensor data represents the environment at a first time; the probability distribution is a first probability distribution associated with a second time after the first time; and the probability distribution is generated prior to the second time. 